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Abstract — A Bayes approach to nonsupervised pattern recognition 
is given where n Z-dimensional vector, samples X h X 2 ,..., X n are 
received unclassified, i.e., any one of M pattern sources coi, co 2 ,..., 
with corresponding probabilities of occurrence Q\ 0 , Qm 0 > 

caused each sample X s , s = 1 , 2 ,..., n. 

The approach utilizes the fact that the cumulative distribution 
function (c.d.f.) of X s is a mixture c.d.f., F(X S ) = A' s |o>;) Q io . 

It is assumed that available a priori knowledge includes knowledge 
of M and the family { F( X s |g>;) } , where F( X s |(o;) is characterized by a 
vector Bi 0 . In general, B io and Qi 0 , i = 1 , 2 ,..., M are considered fixed 
but unknown, and conditional probability of error in deciding which 
source caused X n is minimized. 

When the functional form of F( XJwi) in terms of Bi 0 is unknown, 
the family {F(Z s |<oOJ is taken to be the family of multinomial 
c.d.f.’s — an application of the histogram concept to the nonsuper- 
visory problem. Additional nonparameteric a priori knowledge about 
the family — such as F( X|w ( ) is symmetrical, and/or F(X s \gh) differs 
from F(X,|co,) only by a translational vector — can be utilized in the 
Bayes solution. 

I. Introduction 


A. The Problem 


J 


ET THERE BE M possible pattern sources, 
coi, co 2 , • • • co M , only one of which is active, to 
produce an Z-dimensional vector sample X s at 
the receiver as shown in Fig. 1. It is assumed that if a 
pattern source co* is active on the sth sample, then the 
cumulative distribution function (c.d.f.) of X s is F(X S | o) { ). 
Let P(co t ) = Q io be the probability that the ^th pattern 
source is active causing X s , and assume Q ift fixed — inde- 
pendent of s. 


with minimum risk, as to which pattern source is active 
on this nth sample. We are especially interested in a 
general approach including the case when the family 
{F(X 8 | Wi)j is unknown and the a priori source prob- 
abilities are unknown. 

To help illustrate the problem further, consider the 
following example with two sources (M = 2) and one- 
dimensional vectors at the receiver (l = 1). When source 

is active, assume the waveform at the channel input is a 
constant m lo and when source co 2 is active, a constant 
m 2o . Source is assumed active with probability Q u 
while co 2 is assumed active with probability (1 — Q lo ). 
Assume that the waveform channel simply consists of 
additive, Gaussian white noise with no memory such 
that the probability density function of X 8 , given source 
a)i as active, is Gaussian with mean m io and variance a 0 . 

For this example, the channel model of Fig. 1 reduces 
to that shown in Fig. 2. Experimentally, given X 1} 
X 2) • • 9 , X n , one problem is to “learn” or estimate 
m u , m 2o , oo, and Q lo . Another problem is, given X lt 
X 2 , • • • , X n , to make a decision with minimum prob- 
ability of error as to which source was active to cause 
sample X n . 

It will be advantageous to refer back to the example 
illustrated in Fig. 2 in a later section where computer 
simulated results are presented. Unless otherwise indi- 
cated, however, the model shown in Fig. 1 is under 
consideration. 



Fig. 1. Channel model. 


f ( x s ! V*i 1 = > °o 5 

f ( X 6 l < V ni 2 ] = 


We consider the problem where n vector samples, 
X ly X 2 , • • • , X n (let Y n = Xi, X 2 , • • • X n ) are received 
unclassified (i.e., it is not known at the receiver which 
pattern source is active to produce sample X s , s = 1, 
2, • • • n) and a decision is to be made on the nth sample, 
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Fig. 2. Binary (M = 2) one-dimensional (1 = 1) example. 

B. Approach 

We will call our approach a mixture [10] approach to 
nonsupervisory problems because the c.d.f. of X 3 is 

M 

= ZUU |co,)Q jo . (1) 

i =1 

In order to apply classical Bayes results to the problem, 
we define a “parameter-conditional mixture,” 
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M 


F(X, \ B 0 ) = N W \ u { , B i 0 )Qi, 

i = 1 


( 2 ) 


where B io contains all the parameters characterizing 
F(X S I c Oi) and B 0 contains all parameters in B u , B 2q , • • • , 
B m o and Qi 0} Q 203 ? Qm 0 ’ 


Goal 1 

For the case when F(X S j is unknown, we approxi- 
mate it by a multinomial c.d.f., thus obtaining a set of 
parameters B io which characterizes F(X S | «,•). This 
might be called an application of the histogram concept 
to nonsupervisory problems. We also give a construction 
technique for utilizing such nonparametric a priori knowl- 
edge as F(X S j c o { ) is symmetrical or F(X S j o\-) differs 
from F(X S | w,-) j 7* i only by a translation vector. 

Goal 2 

As defined, B 0 contains all the parameters character- 
izing this nonsupervisory problem, including Q Uy Q 2o , * * * , 
Q Mo . As a second goal, we want to treat all parameters 
in B 0 as fixed but unknown, and show under what sufficient 
conditions all these parameters can be “learned” as well 
as give the minimum conditional probability of error 
solution. Consistent with a Bayes approach, we will let 
B, Bi, and Q i} be random variables corresponding to our 
uncertainty in B 0j B io , and Q ioJ respectively. 


Discussion of Goal 2 

Spragins [17] considered a convergence theorem giving 
sufficient conditions (i and ii below) for a Bayes solution 
to converge. 


1) 



There exists a sequence of functions f n (Y n ) con- 
verging to B 0 with probability one. 


f(B) > 0 in some sphere containing B 0 . 

The a posteriori density f(B j Y n ) is calculated 
by the Bayes rule. 


The solution to the nonsupervisory problem presented 
in this paper is a Bayes solution; consequently ii)2 above 
is satisfied. We assume ii) 1 is satisfied. The approach 
taken in this paper makes it possible to examine sufficient 
conditions for i) to be satisfied. These sufficient condi- 
tions are specified in terms of the a priori knowledge 
defining the nonsupervisory problem. 


a) The number of pattern sources iff 

b) The family {F(X, j w*)} 

c) Any constraints on B or X s sufficient for identi- 
fiability (defined in Appendix I) 

d) Any constraints additional to c) required in order 
to exhibit a consistent estimator for B 0f thus satis- 
fying i) above. 

In summary, the nonsupervisory problem is partly 
defined in terms of a priori knowledge a) and b) . In terms 
of this a priori knowledge, sufficient constraints (if they 
exist) are determined to obtain property c)— identifi- 
ability. Then, further constraints are imposed to obtain 


d)- — i.e., to insure that a consistent estimator for B 0 can 
be found. After all this, we are assured that given a priori 
knowledge a), b), c), and d), the Bayes solution will 
converge. 

In this paper, we will allude to the references con- 
cerning d), a research problem where there are only a few 
results. In Appendix I, we consider sufficient constraints 
c) for identifiability for several nonsupervisory problems. 


G . Literature Review — - Bayes Approach to N onsupervised 
Pattern Recognition 

An optimality criterion frequently used is as follows. 
Given Y n - U make a decision as to which of M pattern 
sources caused sample X n . This decision is made by a 
decision function obtained with a system constraint of 
minimum sample-conditional probability of error. I he 
word sample is used here to make clear that we are talking 
about probability of error conditioned on the past samples, 
V 

J n— 1 • 

Abramson and Braveman [1] considered a problem 
where it is known which pattern source caused samples 
X s , s = 1, 2, • • • , n { (i.e., the samples are classified). 
That is, the a priori knowledge includes knowledge that 


F(X S | B 0 ) = F(X S [ £o <f B io ), i known, s = 1, 2 • • • , w t -. 

Further, it is known that the family {F(X 8 [ co*) } is a 
multidimensional Gaussian family, with only the mean 
vector m io (in B io ) unknown for each member of the 
family. The authors also assume a c.d.f. F(B) is avail- 
able describing the a priori uncertainty in the unknown 
parameters. Using all this a priori knowledge, they ob- 
tained a system minimizing sample-conditional prob- 
ability of error. 

Keehn [15] extended the work of Abramson and Brave- 
man to the case where the mean vector and covariance 
matrix (in B io ) are unknown. He carefully defined c.d.f.’s 
F(B:) for all i such that the a posteriori c.d.f. of B iy for 
each i is reproducing [20]. 

Daly [3] investigated a nonsupervisory system where 
the classification of the samples is unknown. A priori 
knowledge includes: knowledge that there are M pattern 
sources, the family {F(X S | «,-)} is known, the set of 
mixing parameters {Q io }f= 1 are known, and a c.d.f. F(B) 
is available. Daly computed the decision function which 
minimizes the sample-conditional probability of error 
for decision on sample X n . Application of this decision 
function requires computation of f(X n , o) t | Y n - 1 ), i = 1, 
2, • • • M. His computation of f(X n , co t - | Y n ^) is a sum of 
terms, thus requiring rapidly increasing computer 
memory. He did not consider sufficient constraints on the 
parameters for the system to converge. 

Fralick, [2], [14] looking for an iterative solution to 
Daly 's problem, obtained an iterative form for f(Bi j Y n - 1 ) 
assuming that if characterizes F(X S j o) { ) and B io 


1 More generally, sample conditional risk can be minimized. \\ e 
will, for practical reasons, be concerned with sample-conditional 
probability of error. 
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characterizes F(X S | co 3 ), then F(Bi \ F n _ 1? B 3 ) = 
F(£. | F n _i). 

Patrick and Hancock [18], [16] showed more generally, 
without making the assumption F(B { | Y n - U B d ) = 
F(B; | F n _]), that the desired a posteriori probability 
density /(B; [ F n _ i) is either of the growing form or, 
equivalently, is computed by integrating the joint density 
f(B | Y n -i) with respect to all vectors except B iy where 
f(B | F n _i) has an iterative form. 

Some of the first work on applying a histogram, ap- 
proximating F(X S | a?,-), to adaptive communication 
systems was done by Sebestyen [5], [8]. He considered 
only classified samples. Patrick and Hancock [6], [7] 
introduced the concept of a histogram, approximating 
F(X S | co;), to the nonsupervisory problem. They pre- 
sented computer simulated results [7] showing rates of 
convergence for a few examples. 

Teicher [9], [10] defined a mixture c.d.f. and identifi- 
ability, and gave a theorem giving sufficient conditions 
for a mixture to be identifiable. Identifiability assures 
that a unique solution B 0 of F(X a ) = F(X S [ B) exists, 
where F(X a ) is a mixture, and with a few additional 
constraints d), frequently permits exhibition of a con- 
sistent estimator for B 0 . In Appendix I we include and 
give a simple extension of some of Teicher’s work, and 
define a parameter conditional mixture F(X a j B) which 
is a useful concept for applying Bayes Theorem to mix- 
tures. In addition, we state several propositions giving- 
sufficient conditions for a mixture to be identifiable. 

Among others who have been concerned with identifi- 
ability c) and consistent estimators d) are Cooper and 
Cooper [4] and Robbins [12]. 

Scudder [19], [20], Jakowatz, Shuey, and White [21], 
and Proakis and Drouilhet [23] have investigated “De- 
cision Directed Measurement” techniques when samples 
are unclassified. It would be of value to indicate the a 
priori knowledge assumed in the design of such systems 
and to list this a priori knowledge a), b), c), and d) as in 
Section I. B. It may be that such systems are optimum 
against the a priori knowledge they utilize, but do not 
use all available a priori knowledge. A system not using 
certain a priori knowledge may be simplified and better 
in some overall cost consideration even though it does not 
minimize conditional probability of error against all 
available a priori knowledge. 

Among others who have contributed to adaptive com- 
munication systems, Kailath [22] gave an estimator- 
correlator interpretation to a minimum probability of 
error system and extended the interpretation to adaptive 
systems. 

II. Mixtures and a Construction Technique 

A. Mixtures and Parameter Conditional Mixtures 

In this section we show that the c.d.f. of X s , when X s 
is unclassified, is a mixture c.d.f. We then define a param- 
eter-conditional mixture in anticipation of the Bayes 
solution in Section III. 


A mixture results when a vector X s can be partitioned 
M ways, co l7 co 2 , • • • oo M , as illustrated in Fig. 1. For 
example, define by (X a , co,) the event that X s is caused 
by pattern source co; being active. Since only one of the 
M pattern sources can be active to cause X s , there are 
M possible mutually exclusive and exhaustive events, 
(X 8 , coO, (X 8 , co 2 ), ••• , (Xs, co M ). If P(oi) = Qi, inde- 
pendent of s, 

M 

. f(x;) = T,f(x, |«<)q,. (3) 

i = 1 

When we speak of a family of Gaussian c.d.f. ’s or a 
family of multinomial c.d.f.’s, we have in mind the nature 
of the parameters which characterize the family. It is 
therefore appropriate to define a parameter-conditional 
mixture c.d.f. F(X S | B) constructed using the collection 
{F(X S | co;, Bi } where F{X S j co,) is characterized by B { . 
To do this, define 

B = B x O B 2 \J - B m+1 (4) 

where 

B { : vector characterizing F(X S | co»-) 

B { VJ B f : collection of all entries in both B { and B 3 
Bm+i — { Qi 1 ¥ • ( 5 ) 

Thus, B is simply the collection of the mixing parameters 
and all entries in B 1} B 2 , • • • B M . In other words, B con- 
tains all fixed but unknown parameters characterizing 
the problem. 

Since (X a , coO, (X B , co 2 ), * * - , (X a , <a M ) are mutually 
exclusive and exhaustive events, 

M 

F(X S ! B) = Y F(X . 1 B)P(co ; I B). (6) 

i = 1 

Now, F(X S j co;) is completely characterized by B i} 
therefore 

F(X S | CO; , B) - F(X a \c 0;,B;) (7) 

and since B contains Q i} 

P(<*i I B) = P( Wi ) = Qi. (8) 

Thus, (5) becomes 2 

M 

FiX, | B) = Y F(X, | Ut , B,)Qi. (9) 

i = l 

Given F(X a ), when does F(X a ) = F(X a | B) have a 
unique solution B 0 ? The answer is that there is a unique 
solution B 0 when the class of mixtures is identifiable, 
several sufficient conditions for which are given in Ap- 
pendix I. Given identifiability c), a consistent minimum 
distance estimator [18] for B 0 can be exhibited when 
F(X S | co;, B^ is continuous in X a and R;— thus establish- 
ing property d). As stated earlier, with c) and d) satisfied, 
the Bayes solution will converge. 

2 For known parameters, (9) becomes 

F(X, | Bo) = Xf-* F{X, | w<> BiJQi,. 
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In the next section, we give sufficient constraints for 
c) and d) to be satisfied when F(X S j co*) is approximated 
by a multinomial c.d.f. 


B. The Fixed Bin Model 


A priori knowledge of the family { F(X S \ w t ) } is required 
before constructing the parameter-conditional mixture (9) . 
Our purpose now is to apply the histogram concept to 
this nonsupervisory problem, for situations where the 
family {F(X S [ co*)} is unknown. To do this, we develop 
a construction method where a multinomial c.d.f. approxi- 
mates F(X S | co,-), utilizing nonparametric a priori knowl- 
edge about F(X S | co*) for each i. Thus we will have 
achieved the first goal of this paper. 

When samples X l , X 2 , • • • X n are all from the same 
class, say co*, it is well known that a histogram can be 
formed from these samples to approximate f(X s | co*). 
In a nonsupervisory problem where the samples are un- 
classified, a certain sufficient amount of a priori knowl- 
edge is required if a histogram for f(X s j c o*) is to be ob- 
tained. We will now give a construction method which 
leads to a determination of such a sufficient amount of a 
priori knowledge. 

Being more general than in the previous Section, let X, 
be a sequence of v Z-dimensional samples, X Sl , X S3 , - • • X s „ 
with the same pattern class active for all v samples. 

Assume further that, given co* and B i} the v samples 
are statistically independent. In order to keep the same 
notation as in A, let X s = {X 5Jb }J, and let Y n = X 1} 
X 2 , • • • X n . 

X sk is an Z-dimensional vector. W"e quantize each of 
these dimensions into R levels, obtaining R 1 Z-dimensional 
u cubes” or “bins”. Each Z-dimensional bin is assumed to 
have the same volume. X si can lie in any of these R 1 
bins, or in the (R l + l)st bin representing the remaining 
part of the Z-dimensional space. The bins are indexed 
and indicated by 7$, £ = 1, 2, • • • , ( R l + 1). 

F(X e | co*) is now approximated using a multinomial 
c.d.f. characterized by the set of parameters G x — 
(pi, pi, • • • , Pri) where p\ is the probability X sk falls in 
bin J $ , given pattern class co* is active. The probability 
that X sk falls in bin I R i + i— given pattern class co* is 
active (denoted by p\ R i+ 1} ) is given, in terms of p[, • • • , p R i 
by 

R l 

P\r 1 +d = 1 — Pf, i = 1,2, ••• M. (10) 

£ = i 


Since X s is a sequence of v samples, v of the (R l + 1) 
bins receive samples during the sth sequence, not all bins 
being necessarily different. Let this relative frequency 
in the bins during the sth sequence be denoted by 

F s \Psn * t ^S(S ! +1))| ^«f 0,1, * (11) 


The probability distribution of V 8 , given pattern class 
co t and G x , is multinomial: 


P(V S | co*-, G*) = 


v\ 


R l + 1 


U S i • U 8 (R { +i) • f = 1 


n &>«]••* (i 2 ) 


and analogous to (8), 

M 

P(V. \B) = 2>(F 5 \ Ut ,B i )Q i (13) 

i = 1 


where 


B < = G { 

B = (G 1 , (?“, • • • G M , Qi, • • • , Q m )‘ 



An example of the fixed bin model for Z = 1, M = 2, 
and v = 1 is shown in Fig. 3. For this case, (12) and (13) 
combine to give 





i — 1 


£ = 1,2, ••• ,R l + 1 


where 

p\ = P[X ak falls in Bin 7 f ], 


• I 2 

p = Q p +(1 -Q) p 

> 1 1 1 r > 




QUANTIZED 
SAMPLE SPACE 


QUANTIZED 
W, SPACE 



QUANTIZED 
u/ 2 SPACE 


Fig. 3. Quantized spaces, l == 1 7 v = 1, M = 2. 


C . Sufficient a priori Knowledge for Identifiability 

As stated in Section II-A, a nonsupervisory problem 
will be said to be identifiable if, given F(X S ) and the 
parameter conditional mixture F(X S ] B), 

F(X S ) = F{X. | B) (15) 

has a unique solution B 0 . 

It is possible to consider particular examples of non- 
supervisory problems and decide if they are identifiable. 
We will define a particular nonsupervisory problem ac- 
cording to the a priori knowledge required to construct 
F(X a | B) (9) and according to constraints on the param- 
eter space B and/or the sample space X s . That is, a 
particular nonsupervisory problem is defined by 

a) the number of pattern classes M 

b) the family {F(X S | co*)} 

c) any constraints on B or X a sufficient for identifi- 
ability 

d) any additional constraints to insure the existence of 
a consistent estimator for B 0 . 
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If, using a) and b) to construct F(X S \ B) and imposing the 
constraints c), (15) has a unique solution B 0 , then it will 
be said that sufficient a priori knowledge exists for 
identifiability. 

For example, if f(X g 1 <a if B io ) is one-dimensional Gaus- 
sian with B i(t = (m io , (T io ) and M = 2, Proposition 1 
shows that for identifiability it is sufficient that a lo ^ cr 2o , 
or if cr lo = a 2o , m lo ^ m 2a be constraints on B 0 . 

Further, for the binary ( M = 2) on-off case (m 2o = 0 
and this is known), no constraints on B 0 are required. 

For the case where the family {F(X S | co.-) } is multi- 
nomial or where the fixed bin model is used because the 
family is unknown, we are concerned with the param- 
eter conditional mixture (13). Since X s is a discrete 
vector, identifiability requires that 

P(X S ) = P(X S | B) 

have a unique solution B 0 where P (X s | B) is a parameter 
conditional mixture of multinomial distributions. Propo- 
sition 5 shows that, for identifiability, it is sufficient that 
v > 2 M — 1 where v is such that 

= {X sk }Ui. 

This is the reason why, when introducing the fixed bin 
model, we made provision for taking v samples in the 
sth sequence with the same pattern class active. In par- 
ticular, for the binary case (M = 2) it is sufficient to 
take 3 or more samples while the same pattern class is 
active. 

In general, the number of parameters in B { is quite 
large when the fixed bin model is used. This is because 
the fixed bin model assumes little knowledge about the 
family {F{X S j co t ) } — only that F{X S | co,-) can be approxi- 
mated by a multinomial c.d.f. On the other hand, if the 
family \F(X S | a?,-)} is known a priori to be Gaussian, B { 
contains a mean vector and a covariance matrix — usually 
fewer parameters than for the fixed bin model case. This 
would seem to suggest that the less known a priori, the 
more parameters will be required to characterize the 
problem. 

The fixed bin model introduces a construction tech- 
nique for utilizing a priori knowledge. In the following 
section, we will show how such a priori knowledge as 
F(X 8 j o)i) is symmetrical, and/or F(X S | wj differs from 
F(X b | co,-), j 9 * i, only by a translational vector, can be 
taken into account. Utilizing this additional a priori 
knowledge reduces the number of fixed but unknown 
parameters in B. We will also show how to take into 
account a priori knowledge that f(X s | co,-) = 0 for X s 
in a given region of the sample space. This will be done 
for the binary case (M = 2). 

D. Utilizing Additional a priori Knowledge About 
F(X 8 | CO;) 

If it is known, for example, that F(X S [ co,-) is sym- 
metrical, then approximating this c.d.f. by a multi- 
nomial c.d.f. does not utilize the symmetrical knowledge. 
For this case, we would use an appropriately defined 


symmetrical multinomial distribution to approximate the 
c.d.f. If, as another example, it is known that F (.X" s j co,- _) , 
i — 1, 2, • • • , M , differs only by translational vectors, we 
would approximate each F(X S ] co<) by an appropriately 
defined translated multinomial c.d.f. 

We have not yet said how we propose to count the bins 
in Z-dimensional space, although writing pi, pi , • • • p pt 
indicates we must have had some counting procedure in 
mind. One method of counting is to redefine p\ 


Ph 

Then let 


Po. 1 < ja < R for all j a . (16) 


Pi Pi ,i , • • * ,i 


Pr — Pr, 1 , • * -,1 
Pr + i — Pr, 2,i, • • • , i 


(17) 


Denote the family [F(X S | co;, B { )} where B { = G 1 
by fip = {F(X S | co,-, G*)}. This is the family used in 
the construction of the parameter-conditional mixture 
(13). Next, define the family $ TP of multinomial c.d.f/ 
differing only by translational vectors, {0;}. To accom- 
plish this, define Ge 0 where 6 0 is a vector of l indexes. 


dn = 


f R + 1 22 + 1 


R ~b 1 


R odd. 


The vector 6 0 locates the center bin in the Z-dimensional 
space with R L quantum levels used for representing (? 0o . 
In terms of G do , the vector G l characterizing F{X S | co;) 
is expressed as 


G l = G ( eo-ei), with p l R i +1 = 0, i = 1,2, 


, M. 
(18) 


Also define a family 3 S tp of symmetrical multinomial 
c.d.f/s differing only by translational vectors, {#;} by 
letting G S 0 o b e a vector whose entries are symmetrical 
about d 0 in each of the l dimensions. 

Now, as an example, assume that it is known that 
F(X S | co;), i = 1, 2, • • • M are all identical except for 
different translational vectors. We approximate these 
c.d.f/s my members of the family 5> P . 

Accordingly, X s is quantized, and the relative fre- 
quency vector V s is thus obtained. The probability 
density of V s , given B , is (13). 


M 


P(V S IB) = EP(F s | a it B t )Q 


(19) 


2=1 


where 


Bi ■= (Ge 0J e % ) 

R = (Ge of 0i, s Qi , • • • Q m ) 


( 20 ) 


Comparing (20) with (14) we find that the a priori 
knowledge that {F(X S ] co,-)} e reduced the number 
of parameters in B by (M — 1 )R l — M as compared to 
when {F(X S | a?;)} z ff P . If {F(X e | oj { ) } z l J S tp instead 
of T p, the number of parameters characterizing the 
mixture c.d.f. is further reduced by [(R + l)/2 — 1)\ 
for R odd. 
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E. Family of Multinomial C.D.F.’s with Spacial Con- 
straints and v = 1 

Let l = 1 , M — 2, v = 1 , and assume it known a priori 
that F(X S | co 2 ) = 0 for X s < Q x and that F(X S | coj.) = 1 
for X 3 > 0 2 , where 6 1 and 0 2 are translational parameters. 
This “spacial constraint” corresponds to an approxi- 
mation that can be made when the signal-to-noise ratio 
is “sufficiently large,” and F(X S [ co,) is symmetrical 
about its translational parameter 0*. 

Using the fixed-bin model, let {F(X S | w*)} £ $sp, the 
family of symmetrical multinomial c.d.f.’s. Since v = 1, 
all entries in V s will be zero except one entry which will 
be unity. If X s < 0 1} it is classified as from pattern class 
coij if X s > 0 2 , it is classified as from pattern class o> 2 , 
but if 0j < X 8 < 0 2 , it is unclassified. Thus, the prob- 
ability distribution of V 9 , given B , is 

P(V a | COi , BJ, X 3 < 0 ! 

P(V a i B) = J P(V 8 I C0 2 , B 2 ), X 8 > 0 2 (21) 

^P(V S I Ui,Bi)Qi, 0! < X s < 0 2 

>i = l 

where 


It is shown in Patrick [18] that if f(B | F n _ x ) is the 
sample-conditional density of B , then the sample con- 
ditional risk is 

r(d j F„_0 = / dlJj dX n 

M I) 

Z L(d(X n ) I C 0 t )j{X n I co,., B t )Q { >f(B I F„_0. (23) 

*' = 1 JJ 

It is well known that for a 0, 1 loss function, the decision 
function minimizing (23) is 

d(X n ) : choose co,- such that 
/(*., «, I L-O = Sup {f(X n ,coj I F n _i) } . (24) 

i 

For this loss function, minimum conditional risk is 
equivalent to minimum conditional probability of error. 

When X TO is a discrete random vector, the decision 
function for 0, 1 loss function is 

d(X n ) : choose co, such that 

P(X n , co j J F n _0 = Sup {P(X n , co, I F n _0 }. (25) 


Pi = 0<), i = 1,2 

P ~ (Ps,6x j G s ,& 2 > ^1, ^2, Ql) 



and is a vector of probabilities p], • • • , p^ R odd, 
symmetric about the middle entry p\ R+1)/2 . 

An example using the above a priori knowledge where 
samples greater than Q x but less than 0 2 were not used is 
given in Patrick and Hancock [7]. Histograms for F(X S j co x ) 
and F(X a [ co 2 ) were obtained while using sample quantiles 
as estimators for 0 U and 0 2o . In this example, Q lo = \ 
and was known a priori, but 0 lo , 0 2o> F(X S j co x ) and 
F(X S ] co 2 ) were unknown a priori. Average error vs. n 
was obtained by computer simulation with F(X a | co,) 
Gaussian (but unknown at the receiver). This average 
error is shown to converge, for practical purposes, to the 
probability if error obtained when 0 lo , 0 2o are known, 
and F(X a | co,) known Gaussian. Equation (21), how- 
ever, when used in the minimum sample-conditional 
probability of error solution presented next in this paper, 
shows how T to use those samples greater than 0 X and less 
than 0 2 , and does not require external estimators for 
0 lo and 0 2o . 


III. Minimum Conditional Probability of Error 
A. Introduction 

We are interested in observing sample X n and deciding 
which pattern source co, caused X n . For each pattern 
source co, and a given decision function d(X n ), we de- 
fine the conditional loss L(d(X n ) [ co»), independent of B 
and F n _! = {X,}!” 1 . Also, we define the conditional 
risk r(d \ co ,) as the conditional loss averaged over all 
values of X n . 


B. Computation of f(B j F B _0 for Mixtures 

In order to minimize sample-conditional probability 
of error, f(B 1 F n _j) must be computed using a priori 
knowledge a), b), c), and d) of Section II — C and F(B ) — 
at least an appropriately defined uniform c.d.f., not 
ruling out the true value of B. 

Working with density functions rather than c.d.f. 's, 
f(B | F n _x) is given by Bayes Theorem as follows: 



The denominator on the right side of (26) is a normali- 
zation constant which assures that f(B | F n _ x ) integrates 
over the B space to unity. f(B j F n _ 2 ) is the density in 
the B space at the (n — 2) stage, /(X n _ 1 \ B) is a function 
directly utilizing the a priori knowledge above. It can be 
shown that the identifiability requirement c) is sufficient 
to guarantee the existence of an estimator for B 0 , con- 
verging to B 0 with probability one, under rather general 
conditions. For example, Patrick [18] has exhibited a 
minimum distance estimator Z B 0 when F(X a | «<, B,) 
is continuous in X s and B,. Blischke [11] has exhibited 
consistent moment estimators for the parameters of a 
mixture of two (M = 2) binomial c.d.f.’s. His result 
immediately extends to a mixture of two (M = 2) multi- 
nomial c.d.f.’s. 

Further research is required to determine more general 
conditions under which identifiability assures the existence 
of an estimator converging to B 0 with probability one. 
When such an estimator does exist, we are assured that 
f(B j F n _ x ) converges to a Dirac delta function [17] at B 0 . 

Using a priori knowledge a) and b) to construct 
/(X n _ i [ B) according to (9), (26) becomes 
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KB I F n _0 = 


M 

Z 

— i = 1 

^ i i B C , 

KB 1 f„_ 2 ) 

/(X„_i 1 F„_ 2 ) 


(27) 


Equation (27) is a fundamental result for the a posteriori 
probability density of the vector B characterizing the 
parameter-conditional mixture. Note that B contains all 
parameters characterizing this nonsupervisory problem, 
including the mixing parameters {Q,}^. 

Using the iterative solution for f(B | F n _ x ) in (27), the 
decision equation (24) is implemented by computing 

/(X„, a,.- I F n _i) = f Q,/(X„ I co,., B,)j(B ! F n _,) dB (28) 

and the decision equation (25) is implemented by com- 
puting 

P(X„, CO, I F„_i) = J QiP(X n ! co,., b;)S(B I F n _0 dB. (29) 

C, Systems Minimizing Sample Conditional Probability 
of Error 

In this section we consider the design of systems 
minimizing sample-conditional probability of error. When 
F(X S | oo i) is approximated by a multinomial c.d.f. using 
the fixed bin model, X 8 is a discrete random vector. We 
therefore use decision equation (25) with P(X ny o>, | F n „j) 
computed in terms of f(B j Y n ^i) by (29). 

Using the fixed bin model, denote the bins that the v 
samples on the nth observation fall in by I vky k = 1 , 2, • • • , v . 
Using this notation, we obtain from (12) and (29) that 


P(X n , co, i F n _0 


vlf 


n pi (Q. 


L VJfc-l 


KB \ F n _j) dB. (30) 


It is convenient to define the sample-conditional ex- 
pectation of 


vl 


II pU (Qi 


L_ k & = 1 


by a*_ i ; that is 


a 


Ti—l 


E 


v 


! II Plk fQi I Yn-l 


k — 1 


(31) 


If v = 1, (31) reduces to 

aU = ElplQi | F n _j] 


(32) 


where p\ is the probability X n falls in bin I n given that 
co, caused X n . 

Equation (31) used along with decision equation (25) 
has an interesting interpretation: To minimize sample- 
conditional probability of error while making a decision 
on the nth sample, observe the relative frequency vector 
V n . Then compute the probability of this relative fre- 
quency vector given the past samples F n _i and co,, for 
each i y and make decisions as follows: 


a 


n— 1 


choose o)j 
= Sup {a*_i} 


(33) 


When F(X S | a B { ) is continuous in X, and B : , the 
decision equation is (24). For example, if the family is 
multinomial Gaussian, 


/(X n _, !&),., B { ) = 


|$; x n 


- 0,.)} 


•exp {-|(X„_ 1 - e^ T [<tS\x n ., 

where 

4>L = E[X T S X S I co,] and 0, = E[X S | co,]. 

Note that 

= ($1, 0,) 

B = ((Clf=i, , mt i). 

The two types of optimum systems which make deci- 
sions with minimum sample-conditional probability of 
error are shown in Fig. 4. The upper system uses the fixed 
bin model, assuming the family {F{X a | co,)} is multi- 
nomial and the lower system is for cases where the family 
is known and whose £th member is continuous in B { 
and X a . 

Through (27)-(29), we have the minimum sample- 
conditional probability of error solution for this non- 
supervisory problem and have thus obtained the second 
goal of the paper. 


f (B | Yn — I ) 


M 
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lPK)f(x n B ijWi ) 

i=l 


A 


Integote 
with respect 
to B 


) f(XnlYn_|) 
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Fig. 4. Minimum conditional probability of error systems. 

D. Marginal a posteriori Density of a Parameter in B 

Sometimes it is desirable to obtain the a posteriori 
probability of just one parameter in B. The Bayes esti- 
mator of one parameter, for example, is calculated from 
the marginal a posteriori probability density of that 
parameter. Therefore, let y kj be some parameter in B k . 
The sample-conditional density of y kj is obtained by 
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E. Family of Multinomial C.D.F.’s with Spacial Con- 
straints and v = 1 

Let l = 1, M = 2, v = 1, and assume it known a priori 
that F(X S j co 2 ) = 0 for X s < 6 1 and that F(X S | co x ) = 1 
for X* > 0 2 , where 6 l and 0 2 are translational parameters. 
This “spacial constraint” corresponds to an approxi- 
mation that can be made when the signal-to-noise ratio 
is “sufficiently large,” and F(X S | co,) is symmetrical 
about its translational parameter 0*. 

Using the fixed-bin model, let {F(X S j co*)} £ ^p, the 
family of symmetrical multinomial c.d.f.’s. Since v = 1, 
all entries in V s will be zero except one entry which will 
be unity. If X a < 6 lf it is classified as from pattern class 
co x ; if X 8 > 0 2 , it is classified as from pattern class co 2 , 
but if 0! < X 8 < 0 2 , it is unclassified. Thus, the prob- 
ability distribution of V a , given B, is 


P(V. | B) = 




P(V 8 | C*?! , Bi), 
P(V S |c 0 2 ,5 2 ), 

tf(F. |< c t ,B t )Q i: 
. » = 1 


X 3 < 0 X 
X s > 0 2 

01 < X a < 0 2 



where 


It is shown in Patrick [18] that if f{B j F n _i) is the 
sample-conditional density of B, then the sample con- 
ditional risk is 

r(d | F n _0 = f dslf dX n 

M 

■ X L(d(X n ) I «,)/(*» I £,)Q,- 

— i = l 

It is well known that for a 0, 1 loss function, the decision 
function minimizing (23) is 

d(X n ) : choose co f such that 


>f(B | F n _i). (23) 


f(X n , co, j F n _0 = Sup {f(X nt co, | F n _0} . (24) 


For this loss function, minimum conditional risk is 
equivalent to minimum conditional probability of error. 

When X n is a discrete random vector, the decision 
function for 0, 1 loss function is 

d(X n ) : choose co, such that 

P(X n , co, I F n _0 = Sup {P(X n , CO, I F„_i)} . (25) 

i 


B , — (Gs,dn 0*) , i — 1,2 ^2) 

B = (G s ,e 1 j Gs,e 2 , 0i, 02, Qi) 

and is a vector of probabilities p], • • • , p^, R odd, 
symmetric about the middle entry p\ R+1 )/ 2 - 

An example using the above a priori knowledge where 
samples greater than 0 X but less than 0 2 were not used is 
given in Patrick and Hancock [7]. Histograms for F(X S | wO 
and F(X a | co 2 ) were obtained while using sample quantiles 
as estimators for 0 lo and 0 2o . In this example, Q lo = J 
and was known a priori, but 0 U , 0 2o , F(X S | co x ) and 
F(X 3 | co 2 ) were unknown a priori. Average error vs. n 
was obtained by computer simulation with F(X a | co,) 
Gaussian (but unknown at the receiver). This average 
error is shown to converge, for practical purposes, to the 
probability if error obtained when 0 lo , 0 2o are known, 
and F{X a j co,) known Gaussian. Equation (21), how- 
ever, when used in the minimum sample-conditional 
probability of error solution presented next in this paper, 
shows how to use those samples greater than 0 X and less 
than 0 2 , and does not require external estimators for 
0 lo and 0 2o . 

III. Minimum Conditional Probability of Error 
A. Introduction 

We are interested in observing sample X n and deciding 
which pattern source co , caused X n . For each pattern 
source co* and a given decision function d(X n ) ) we de- 
fine the conditional loss L(d(X n ) | a > t ), independent of B 
and F n _! = {X*}"” 1 - Also, we define the conditional 
risk r(d [ co,) as the conditional loss averaged over all 
values of X n . 


B. Computation of f(B j F n _ 1 ) for Mixtures 

In order to minimize sample-conditional probability 
of error, f(B | F n _j) must be computed using a priori 
knowledge a), b), c), and d) of Section II — C and F(B ) — 
at least an appropriately defined uniform c.d.f., not 
ruling out the true value of B. 

Working with density functions rather than c.d.f. ’s, 
f(B | F n _i) is given by Bayes Theorem as follows: 

j(T) j y \ _ /(Xn-! 1 B) (B [ F w - 2 ) , op\ 

j(B ! F,_.) - J (26) 

The denominator on the right side of (26) is a normali- 
zation constant which assures that f(B | F„_i) integrates 
over the B space to unity. f(B | F n _ 2 ) is the density in 
the B space at the (n — 2) stage. f(X n ^ 1 j B) is a function 
directly utilizing the a priori knowledge above. It can be 
shown that the identifiability requirement c) is sufficient 
to guarantee the existence of an estimator for 5 0 , con- 
verging to B 0 with probability one, under rather general 
conditions. For example, Patrick [18] has exhibited a 
minimum distance estimator Z B 0 when F(X 8 j co i} B ,) 
is continuous in X 8 and B,. Blischke [11] has exhibited 
consistent moment estimators for the parameters of a 
mixture of two (M = 2) binomial c.d.f.’s. His result 
immediately extends to a mixture of two (M = 2) multi- 
nomial c.d.f.’s. 

Further research is required to determine more general 
conditions under which identifiability assures the existence 
of an estimator converging to B 0 with probability one. 
When such an estimator does exist, we are assured that 
f(B | F n _x) converges to a Dirac delta function [17] at B 0 . 

Using a priori knowledge a) and b) to construct 
/(X n _ 1 | B) according to (9), (26) becomes 
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rH 

1 

s 

*WI 

1 J 

w,, B t )Qi 

| 

f(B \ 

i F„_ 2 ) 

1(1,-, 

\ f„_ 2 ) 


(27) 


When F(X S | a>,-, B { ) is continuous in X 8 and B i: the 
decision equation is (24). For example, if the family is 
multinomial Gaussian, 


Equation (27) is a fundamental result for the a posteriori 
probability density of the vector B characterizing the 
parameter-conditional mixture. Note that B contains all 
parameters characterizing this nonsupervisory problem, 
including the mixing parameters 

Using the iterative solution for f(B | Y n - X ) in (27), the 
decision equation (24) is implemented by computing 

f(X n , co, I F n _x) = / Q4(X n I co,., I F_0 dB (28) 


/(X„_x jco„£,) = 


1 


((2tt) 7 |$Lr ) 


exp {-KV.-X - e i ) T [<l>i x ]~\X n - 1 - <?,)} 


where 


= E[XlX, | co,] and 0, = E[X. \ co,-] 

Note that 

B { = (4>L, 0 t ) 


and the decision equation (25) is implemented by com- 
puting 

P(X n , CO, I F n _ a ) = / QiP(X„ | co, , I F n _,) (29) 

C. Systems Minimizing Sample Conditional Probability 
of Error 

In this section we consider the design of systems 
minimizing sample-conditional probability of error. When 
F(X 8 | oji) is approximated by a multinomial c.d.f. using 
the fixed bin model, X 8 is a discrete random vector. We 
therefore use decision equation (25) with P(X n) co, | F n „i) 
computed in terms of f{B ] F n _i) by (29). 

Using the fixed bin model, denote the bins that the v 
samples on the nth. observation fall in by I vky k = 1, 2, • * • , v. 
Using this notation, we obtain from (12) and (29) that 


P(X n , C Oi F n _! 


= vlf 


n pu \Qi 


L = 1 


f(B I Y n -i) dB . (30) 


It is convenient to define the sample-conditional ex- 
pectation of 


vl 


an- 


ti pU fQi 


by a*_i;thatis 


'n— 1 


E 


v 


! n pU fa I 


k = l 


(31) 


If v = 1, (31) reduces to 

(in-i = E\p{Q t I F„_i] 


(32) 


where p\ is the probability X n falls in bin I n given that 
o 0 i caused X n . 

Equation (31) used along with decision equation (25) 
has an interesting interpretation: To minimize sample- 
conditional probability of error while making a decision 
on the nth sample, observe the relative frequency vector 
V n . Then compute the probability of this relative fre- 
quency vector given the past samples Y n „ x and for 
each i, and make decisions as follows: 


choose to,- 

a? n -i = Sup {ai-i} • 

i 



b = {QAtu {QAU- 

The two types of optimum systems which make deci- 
sions with minimum sample-conditional probability of 
error are shown in Fig. 4. The upper system uses the fixed 
bin model, assuming the family {F{X 8 | to,-)} is multi- 
nomial and the lower system is for cases where the family 
is known and whose fth member is continuous in B { 
and X 8 . 

Through (27)-(29), we have the minimum sample- 
conditional probability of error solution for this non- 
supervisory problem and have thus obtained the second 
goal of the paper. 


f ( B | Yn — i ) 



Fig. 4. Minimum conditional probability of error systems. 


D. Marginal a posteriori Density of a Parameter in B 

Sometimes it is desirable to obtain the a posteriori 
probability of just one parameter in B. The Bayes esti- 
mator of one parameter, for example, is calculated from 
the marginal a posteriori probability density of that 
parameter. Therefore, let y ki be some parameter in B k . 
The sample-conditional density of y kj is obtained by 
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integrating (27) with respect to all parameters in B not 
equal to y kj . Integrating (27) in this fashion gives 


Ibfki ! Yn- 1 ) 


E ( Q,1(X n -i | B;, i F„_0 dB 

i^k ** 


/(X_i ! f„_ 2 ) 

[ Q k f(X n - x j B k , u k )j{B \ F n _ 2 ) 

H “fTr Tf i G 4 ) 

where B is defined as the vector not containing param- 
eter y kj but containing all other parameters in B. 

Using the fact that 


/(X n _i j > ^ i ) /(A n _i ) ' Ykj ) 


and 


Qkfi^-n— 1 j j ^&) / (A n— 1 j | i 'Ykj) 2 ) 

and defining “weighting coefficients,” 

jf(X/— 1 , | F n — 2? 7fcj>) 


<M7*,) 

(34) becomes 


/(I,-, | F n _ 2 ) 


/fo, I IV-i) = 


^*(7*/) + C k (y kj ) 


L i^k 


E\f(X n - u a, k 

1 

1(Xn-D ^k | 

7 kj ) F n _ 2 ) 


/(t*i I -F,- 2 ) (35) 


where 


iS[/(X B _ 1; CO, It,,., f„_ 2 )] 

— J' /(i-ii ? 7ki)f(B | , F n _ 2 ) 


In general, however, it is necessary to store j(B | F n _i) 
in memory, as indicated above, if digital techniques are 
used. To do this, denote the number of scalar entries in 
B by q and write 


B = (0i , ^ 


2? 


» 0«)- 


(36) 


Quantize into N { one-dimensional segments of length 
A,- each, i = 1, 2, • • • , q. There are then XX ? - x N%. cubes, 
each g-dimensional. Denote a particular cube by 
Lj x , ,- 2 «, and denote the fixed but unknown prob- 
ability measure in this cube by Denote the a 

posteriori probability measure in this cube at the nth 
stage by (m ilf , *,,... ,*„)». Then, assuming /(X, | 5) is con- 
stant as B varies within a cube, we approximate (27) by 


(jYlj 1 * / 2 .***'»/ 5) » 

/ (X n _ 1 [ Lj j , 3 , , • • • , j j (m ,• !,)■ 2 ,••*,} ? ) 1 


q iVi- 

^ ^ > /(X n _i | Lj 1 , ,- a , . . . , ,* ff ) (yyijj , j 2 , • • • , j 5 )n-! 

i—l j » = 1 


(37) 


We have investigated (37) as an approximation to (27) 
using the IBM 7094 Computer. The example reported 
here is a binary (M = 2), one-dimensional example 
(see Fig. 2) where the family {F(X S j co,-) } is known 
Gaussian. A uniform random number generator was used 
to generate the mixing parameter Q lo . If the number 
from the uniform random number generator was < Q lo , 
mean 0 U was added to a number generated from a Gaus- 
sian random number generator with zero mean and 
variance <r 0 . If the number the uniform random number 
generator was > Q lo , then mean 0 2o was added to the 
number from the Gaussian generator. 

For this example, 


The interpretation of (35) is as follows: 

a) Ei** Ci(y kj ) is the probability, given y ki and F„_ 2 , 
that pattern source co k did not cause X n _!. With prob- 
ability E i^k Ci{y k j)y f(7kj I F n _ 2 ) is retained at the 
(n — l)st stage. 

b) C k (y kj ) is the probability given y k . and F n _ 2 , that 
pattern source co k caused X n . With probability C k , 
/( y k . I F n _ 2 ) is updated, assuming X n is caused by pattern 
source cc k . 

c) JS7[/(X n _i, w k 1 y ki , F n _ 2 )] is involved in (35) because 
/(X n _!, co k | B k ) is, in general, a function of parameters 
other than y kr 

IV. Implementation and Computer Simulation 

A . Quantizing the Parameter Space 

The computation of f(B | Y n ) is iterative, in terms of 
f(B | Y n -i). The procedure is that, upon receiving X w , 
f(B 1 F w „ 1 ) is replaced in storage by j(B | F n ). To store 
f(B j F n _i) in memory as mass density in the parameter 
space, it is necessary that B take on a finite number of 
points.- For some cases, it is not necessary to store 
F(B | Fn-0 this way; instead, a sufficient statistic is 
retained [17]. 


B 1 = (0i, a) 

B 2 = (0 2 , cr) 

B — (0i , 02? < 7 , Q) 

i/y I r? \ 1 [”(X S — Bi ) 3 /2<r 2 ] 

/(A s | B i) /— e 

v 2 tt a 

The fixed but unknown vector B 0 is 

Bq = (01., 02 0 ? a o 1 Qlo)* 

And according to Proposition 1, a sufficient constraint 
for identifiability is 0 2o > 0i o . 

First, computer simulated results were obtained for 
average error, using decision equation (24) and (37) for 
only 0 2 0 and 0 lo unknown. For <r 0 — 1, Q lo = J and both 
known, and the constraint 0 2o > 0 io and 90 segments 
of length 1/10 in each dimension of the parameter space 
(8100 two-dimensional cubes, 5040 having zero measure 
because of the constraint 0 2o > 0i o ), the average error is 
plotted vs. n in Fig. 5 for the following 3 cases: 

Case 1: 0 lo = 0, 0 2o = 2.4, and F(() h 0 2 ) uniform — i.e., 
the same measure is put in each of the 3960 cubes to 
begin the iteration. 

Case 2: Q u = 0, 0 2a = 0.5, and F(0t, 0 2 ) uniform. 
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0 12 3 4 5 6 7 8 9 10 II 12 13 14 15 

Fig. 5. Computer simulated average error vs. n for binary Gaussian 
example with two unknowns, 6 lo and 6 2q . 


Average Error 



Fig. 6. Computer simulated average error vs. (6 2q — 0 lo ) for binary 
Gaussian example with three unknowns, 0i o , 0 2o , and Q lo . 


Case 3: 0 U = 0, d 2o = 2, and /(0 1? 0 2 ) = (1/2 tt) exp 
(0! - 1/2) 2 exp (0 2 - 5/2) 2 . 

Case 3 has unfavorable a priori knowledge. 

Second, <r 0 , 9 lo , and 0 2o were unknown at the receiver. 
For Q u = \ and the constraint 0 2o > 6 lo and 45 segments 
along the 6 1 and 0 2 axis and 10 along the a axis, all of 
length 1/10, and with F(d lf d 2 , a) uniform, average error 
vs. (0 2o — 0io) plotted in Fig. 6 for two cases: 

Case 1: n = 20, 10 experiments, 9 U = 0, a 0 = 1, 0 2o 
variable. 

Case 2: n = 50, 10 experiments, 0 lo = 0, a 0 — 1, 0 2o 
variable. 

There was a total of 20 250 cubes in the parameter 
space, with zero measure in 10 570 cubes because of the 
constraint d 2o > 0 1q . 

V. Conclusions 

The approach taken in this paper to nonsupervised 
adaptation begins by expressing the c.d.f. of a sample X a 
as a mixture c.d.f. which is given in terms of members of 
a family \F(X a | co t -)} and mixing parameters 

As a first result, for the case when F(X S \ a u) is un- 
known, we approximate it by a multinomial c.d.f. under 
the frame work of a “ fixed bin” model. This can be con- 
sidered an application of the histogram concept to non- 


supervisory problems. The resulting construction tech- 
nique provides for taking into account such nonpara- 
metric a priori knowledge as F{X S ] w*) is symmetrical 
and/or differs from F{X S | w,-), j ^ i, only by a trans- 
lational vector. 

As a second result, we computed the a posteriori prob- 
ability f(B | F n _i) iteratively where B contains param- 
eters characterizing each F(X S | co*) and the mixing 
parameters {Qi\ Systems which minimize sample- 
conditional probability of error were then obtained for 
both the case where the family {F(X S ] co/) } is known 
and the case where the fixed bin model is used. For com- 
puter implementation of both cases, the parameter space 
containing B was quantized. 

The concept of identifiability is introduced to mean 
that, given F{X S ) } B 0 is a unique solution of F(X S ) = 
F(X S | B) where F{X a | B) is constructed from a priori 
knowledge of M and the family {F(X S | co,)}. By alluding 
to the references, it was shown that an estimator con- 
verging with probability one to B 0 can be constructed 
under rather general conditions when given identifiability. 
Then there is a guarantee that f(B | Y n -^) converges 
with probability one to a dirac delta function at B 0 ; 
and the sample conditional probability of error converges 
to the probability of error obtained when B 0 is known. 

Finally, a binary (. M = 2) one-dimensional example is 
given where the family {F(X S | co*)} is Gaussian. By 
quantizing the parameter space, thus approximating the 
Bayes solution, computer simulated results were pre- 
sented for average error vs. the number of samples. 

Appendix I 

Mixtures and Identifiability 

Following TeicheFs definition [10] of identifiability 
for one-dimensional mixture c.d.f. J s, we give the following 
definition of identifiability for ^-dimensional mixture 
c.d.f. ? s. All parameters in this Appendix are fixed param- 
eters and not to be considered variable. 

Identifiability of Mixture C.D.F Is 

Let $ = {F{X | a) : a z R\) constitute a family of 
^dimensional index-conditional c.d.f. 7 s indexed by a 
point a in a subset R\ of Euclidean k- space R k . Then, 
the ^-dimensional mixture c.d.f. 

F(X) = / F(X | a) dG (a) (38) 

is the image under the above mapping, say 5, of the 
/c-dimensional c.d.f. G (where the measure jjl g induced by 
G assigns measure one to Rfi). 

The c.d.f. F(X) is called a mixture (or (7-mixture of Jy) 
while G is referred to as the mixing c.d.f. Let Q denote 
the class of all such c.d.f. ? s G, and 5C the induced class of 
mixtures F(X) (given a priori the family CF). Then 3C 
will be said to be identifiable if § is a one-to-one map of 
g onto 3C. 

F(X) is called a finite mixture if its mixing distribu- 
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tion G. or rather the corresponding measure / u G , is discrete 
and doles out positive mass to only a finite number (M) 
of partitions in R\. Let these partitions be co i} i — 1 , 
2, • • • , M, and the corresponding mass or measure be 
P(co,-) = Qi, i = 1, 2, • • • M. Then (38) becomes 

M 

F(X) = J2F(X \^)Q,. (39) 

i — 1 

t 

C.D.F ds Characterized by Finite Number of Parameters 

Let F(X) be characterized by a finite size vector set 
of parameters B and F(X \ co,-) be characterized by B { . 
Then 

M 

F(X 1 B) = y F(X I co,,, B,)Qi (40) 

r = 1 

B = B X \J B 2 VJ • • • VJ B m \J {Q, }f . (41) 

Then Q is the class of all such vectors B } and 3C the 
induced class of parameter-conditional mixtures F(X | B). 
According to the previous definition, 3C is said to be 
identifiable if fi is a one-to-one map of B onto 5FC. 

Thus, for a given c.d.f. F(X), given the family 
[F(X | co,-)}, M, and identifiability, there is a unique 
vector B 0 such that F(X) — F(X \ B 0 ). 


b) if k is the smallest index such that a k — o* +1 , then 
m k < m k+ i 

c) repeat a) and b) starting with a k+1 < a k+2 , etc. 

In other words, a) • • • c) is sufficient a priori knowledge 
to assure identifiability. It is not the necessary a priori 
knowledge to assure identifiability. We can view a) • • • c) 
as a constraint of the domain of definition of B. If this 
constraint is utilized, then a unique solution for B 0 can 
be found, given the sequence of samples Y n „ 1 asn-> oo . 

For the binary, Gaussian, on-off case, no constraints 
are needed, if a = <r l = a 2 and P 1 are unknown: 

Proposition 2: The class of mixtures of two (M = 2) 
one-dimensional normal c.d.f. ’s, 

F(X | co 1? 0 1} a), F(X J co 2 , 0 2 , <r), 

with cr, Ox, and Q x known, is identifiable. 

The following is a proposition where we have extended 
Proposition 1 to the multidimensional case. 

Proposition 3: Let {F(X [ co,)} be a finite family of 
Ldimensional normal c.d.f. ’s with B i = (0,-, $ % xx ) with 
mean vector 0< = (0,- 1? d ia , • • • 0 i4 ) and covariance matrix 
— [<r)k]- If the family is ordered lexicographically so 
that Ni < N 2 < Na < • • • < N M if al lf • • • , (J k kk > 

* * * , or if cr kk = cr^ 1 but m kk < m k+lik , then the 
family is identifiable. 


Sufficient a priori Knowledge for Identifiability 

We are interested in determining sufficient amounts 
of a priori knowledge for identifiability. The following 
theorem and propositions, proven in Patrick [18], provide 
for determining if the a priori knowledge given in a 
particular nonsupervisory problem is sufficient for identifi- 
ability. 

Theorem 1 


Let = {F(X j co,-,) } be a family whose Ah member is 
characterized by B { with transform # • , v t [ Bfi 

defined for V = (v 1} • • • , v t ) e (the domain of defi- 
nition of such that the mapping A : F — > <f> is linear 
and one-to-one. Suppose that there exists a total order- 
ing (<) of such that F x < F 2 implies: i) Q S 03 , 
ii) the existence of some V x e (Fi being independent 
of <p 2 ) such that 


lim 

F-»Fi 


02(F) 

*i(V) 



Then the class 3C of all finite mixtures of ^ is identifiable. 

A special case of Theorem 1 is 

Proposition 1: The class of one-dimensional, mixtures 
of normal c.d.f. ? s — with constraint that the family be 
ordered lexicographically by N t < N } - if a { < a j or if 
but di < 6j — is identifiable. 

The significance of Proposition 1 is that if the family 
{F(X | oof) } is one-dimensional Gaussian, then, given 
F(X j B), there is a unique solution for B 0 if the a priori 
knowledge includes 

a) ai > cr,*, i < j or 


Identifiability and the Fixed Bin Model 

The family {F{X j co,)} defined using the Fixed Bin 
Model has members which are multinomial distributions. 
In general, mixtures of multinomial c.d.f.’s are not 
identifiable because they are, in general, used to approxi- 
mate c.d.f.’s about which little is known a priori. We 
then ask what constraints must be imposed on the multi- 
nomial c.d.f.’s to insure identifiability? The following 
propositions give a partial answer to this question. 

Let the c.d.f. of X be a mixture of binomial c.d.f.’s, 
F(X | co,) characterized by v and, say, p[. The c.d.f. of 
X is a mixture c.d.f., the corresponding parameter-con- 
ditional mixture c.d.f. being 

M 

F(X I B)= y F(X I V, p\ co,)Q„ (42) 

i = 1 

The question is, what are sufficient conditions under 
which p\ and Q i} i — 1, 2, • * • M can be uniquely found? 
The following Proposition 4 by Teicher [10] gives such 
sufficient conditions. 

Proposition 4: Let {F(X | v, p{), 0 < p\ < 1} constitute 
a one-parameter family of binomial distributions, v 
being fixed and known. A necessary and sufficient con- 
dition that the class Lfc. of all finite mixtures of at 
most M elements of fi be identifiable is that v > 2 M — 1. 

The significance of Proposition 4 is illustrated by the 
binary (M = 2) one-dimensional example of Fig. 1. In 
this example, p\ and p\ can be uniquely found if X 9 con- 
sists of at least three samples from the same pattern 
class. 

We will now give an extension of Proposition 4 to a 
mixture of multinomial distribution. 
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Proposition 5: Let = { F(X S | v, {p\} f =1 ) 0 < p\ < 1} 
constitute a family of multinomial distributions, v being 
fixed. A sufficient condition that the class {Jf JC,- of all 
finite mixtures of at most M elements of fF be identifiable 
is that v > 2 M — 1. 


Symbol 


Symbols 

Description 


X s sth 1-dimensional vector sample 

X s = {X s& }[ sequence of v samples taken at the sth 

observation 

oo i fth pattern source 

F(X S ) cumulative distribution function (c.d.f.) of 


X s 


F(X S | 


M 

1 


fth source-conditional c.d.f. 

F(X S j c B { ) ith source, parameter-conditional c.d.f. 
F(X § | B) parameter-conditional c.d.f. of X s 

number of pattern sources 
M pattern source probabilities — also called 
mixing parameters 

Set of vector parameters characterizing 

F(X S | co,) 

b m+ i = m? 

collection of all vectors in B 1 , • • • , B M , B M +1 
a priori c.d.f. for B 
B io , B o true vectors — fixed but unknown 


M 

m 

Bi 


M + l 


B 
B 

F(B) 


R 

U 


•s 


TP 


^ S TP 

V, 


Fixed Bin Model Notations 

number of quantizing levels for each di- 
mension of X, 

bin J, one of the ^-dimensional cubes in the 



sample space 

v\ 

probability X sk falls in bin I $ 


a ? i caused X 8& 

a 

= (pi pi, ■■■ , pb>) 

3 > 

family of multinomial c.d.f. ’$ 


family of multinomial c.d.f.’s differing only 
by translational vectors 
family of symmetrical multinomial c.d.f. ? s 
differing only by translational vectors 
vector of relative frequency in the It 1 bins 
due to sample X s = {X 8£ |J; 


V, 


(v t 


l ? 


V 


s R 


')■ 
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